Image-capturing device

ABSTRACT

An image-capturing device includes: a plurality of micro-lenses disposed in a two-dimensional pattern near a focal plane of an image forming optical system; an image sensor that includes a two-dimensional array of element groups each corresponding to one of the micro-lenses and made up with a plurality of photoelectric conversion elements which receive, via the micro-lenses light fluxes from a subject having passed through the photographic optical system and output image signals; and a synthesizing unit that combines the image signals output from the plurality of photoelectric conversion elements based upon information so as to generate synthetic image data in correspondence to a plurality of image forming areas present on a given image forming plane of the image forming optical system, the information specifying positions of the photoelectric conversion elements output image signals that are to be used for generating synthetic image data for each image forming area.

This is a Continuation of application Ser. No. 14/575,164 filed Dec. 18,2014, which is a continuation of application Ser. No. 13/474,189 filedMay 17, 2012, which in turn is a continuation of InternationalApplication No. PCT/JP2011/062828 filed Jun. 3, 2011, which claims thebenefit of U.S. Provisional Application No. 61/487,427 filed May 18,2011. This application also claims priority from Japanese ApplicationNo. 2010-127825 filed Jun. 3, 2010. The disclosure of the priorapplications is hereby incorporated by reference herein in its entirety.

BACKGROUND 1. Field of the Invention

The present invention relates to an image-capturing device capable ofgenerating a synthetic image.

2. Description of Related Art

Japanese Laid Open Patent Publication No. 2007-4471, US 2007/0252047 and“Light Field Photography With a Handheld Plenoptic Camera, Stanford TechReport CTSR 2005-02” disclose image-capturing devices known in therelated art that are equipped with a plurality of image-capturing pixelsdisposed in correspondence to each micro-lens, and are capable ofgenerating an image assuming any desired focus position following aphotographing operation by combining image data having been obtainedthrough the single photographing operation.

SUMMARY OF THE INVENTION

There is an issue to be addressed with regard to such image-capturingdevices in the related art in that the image generated as describedabove assumes a resolution matching the number of micro-lenses disposedin the array, which is bound to be much lower than the density withwhich the image-capturing pixels are arrayed. In addition, thearithmetic processing required to generate the synthetic image is boundto be extremely complex.

According to the 1st aspect of the present invention, an image-capturingdevice comprises: a plurality of micro-lenses disposed in atwo-dimensional pattern near a focal plane of an image forming opticalsystem; an image sensor that includes a two-dimensional array of elementgroups each corresponding to one of the micro-lenses and made up with aplurality of photoelectric conversion elements which receive, via themicro-lenses light fluxes from a subject having passed through thephotographic optical system and output image signals; and a synthesizingunit that combines the image signals output from the plurality ofphotoelectric conversion elements based upon information so as togenerate synthetic image data in correspondence to a plurality of imageforming areas present on a given image forming plane of the imageforming optical system, the information specifying positions of thephotoelectric conversion elements output image signals that are to beused for generating synthetic image data for each image forming area.

According to the 2nd aspect of the present invention, it is preferredthat in the image-capturing device according to the 1st aspect, theplurality of image forming areas are set in a quantity equal to orgreater than a quantity of micro-lenses and an array pitch with whichthe individual image forming areas are set has a proportional relationto an array pitch with which the plurality of micro-lenses are disposed.

According to the 3rd aspect of the present invention, it is preferredthat in the image-capturing device according to the 1st aspect, theinformation specifying positions of the photoelectric conversionelements output image signals that are to be used to generate syntheticimage data in each of the image forming areas is configured to a tabledetermining specific positions of the photoelectric conversion elementsthat output the image signals to be used for generating synthetic imagedata for each image forming area

According to the 4th aspect of the present invention, theimage-capturing device according to the 3rd aspect may further comprisea creation unit that creates the table for each given image formingplane.

According to the 5th aspect of the present invention, it is preferredthat in the image-capturing device according to the 3rd aspect, thetable standardizes the positions assumed by the photoelectric conversionelements corresponding to each image forming area in reference to apseudo-optical axis of the micro-lenses and specifies relative positionsof the micro-lenses corresponding to the positions assumed by thephotoelectric conversion elements in reference to the micro-lenscorresponding to the image forming area.

According to the 6th aspect of the present invention, it is preferredthat in the image-capturing device according to the 3rd aspect, thetable determines the specific positions of the photoelectric conversionelements, each in correspondence to a specific micro-lens among theplurality of micro-lenses, to which the position assumed by aphotoelectric conversion element among the photoelectric conversionelements present in an area assuming a diameter represented by a valueobtained by dividing a focal length of the micro-lenses by a syntheticimage data aperture number in reference to the image forming area,corresponds.

According to the 7th aspect of the present invention, it is preferredthat in the image-capturing device according to the 1st aspect, theplurality of micro-lenses each assume a hexagonal shape on a planeranging perpendicular to an optical axis of the photographic opticalsystem and are disposed in a two-dimensional honeycomb array.

According to the 8th aspect of the present invention, theimage-capturing device according to the 7th aspect may further comprisea converting unit that converts a ratio of a horizontal pitch and avertical pitch of the synthetic image data generated by the synthesizingmeans, to 1.

According to the present invention, a table indicating the positions ofspecific photoelectric conversion elements, image signals from which areto be used to generate synthetic image data corresponding to one imageforming area among a plurality of image forming areas on each imageforming plane, is created and synthetic image data are generated basedupon the table thus generated. As a result, synthetic image data withhigh resolution can be generated quickly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the structure adopted in the digitalcamera achieved in an embodiment of the present invention;

FIGS. 2A and 2B are an example of a positional arrangement that may beassumed for the micro-lenses and the image sensor, with FIG. 2Apresenting a plan view taken over the XY plane and FIG. 2B representingthe positional relationship along the optical axis (along the z-axis) ofthe photographic lens;

FIG. 3 illustrates the positional relationship between a micro lens andthe corresponding cardinal point pixel assumed in the embodiment;

FIGS. 4A and 4B illustrate the principal concept of synthetic imagegeneration, with FIG. 4A illustrating that the focal plane is present atthe vertex of the micro-lens and FIG. 4B illustrating that the focalplane is set apart from the vertex of the micro-lens;

FIG. 5 illustrates the relationship between the integration area overwhich image signals are integrated for purposes of synthetic imagegeneration and image capturing pixels;

FIGS. 6A and 6B illustrate examples of positional relationships that maybe assumed with regard to the positions of the image-capturing pixelsthat output image signals to be integrated with a cardinal-point signal,with FIG. 6A illustrating the relationship that may be assumed when thefocus position of the synthetic image is further toward the subjectrelative to the micro-lens array and FIG. 6B illustrating therelationship that may be assumed when the focus position of thesynthetic image is further toward the image sensor relative to themicro-lens array;

FIG. 7 illustrates an example of a relationship that may be assumedbetween the zone and the micro-lenses;

FIG. 8 illustrates an integration area assumed when the cardinal-pointpixel is offset from the pseudo-optical axis of the micro-lens;

FIGS. 9A and 9B illustrate variations in micro-lens shapes andcardinal-point pixel positions;

FIG. 10 illustrates light sections of a light flux which, afterdeparting a light point is sliced off at the light-receiving surfaces ofimage-capturing pixels;

FIG. 11 illustrates relationships between micro-lenses and lightsections;

FIGS. 12A and 12B illustrate a relationship between micro-lenses andlight sections;

FIG. 13 illustrates light sections resulting from the area division ofthe area covered by a cardinal point micro-lens;

FIG. 14 is an illustration of divided areas formed in correspondence toa light point off-centered relative to the pseudo-optical axis of acardinal point micro-lens;

DESCRIPTION OF PREFERRED EMBODIMENT

The digital camera achieved in an embodiment of the present invention iscapable of generating image data assuming a field depth and a focusposition desired by the user through numerical processing executed byutilizing wavefront information such as depth information included inimage signals obtained as an image is photographed via a micro-lensarray. An incident subject light flux, having passed through aphotographic lens forms an image near the micro-lens array. The positionat which the image is formed with the light flux in this manner variesalong the optical axis of the photographic lens depending upon theposition of the subject. In addition, subject light fluxes from athree-dimensional subject do not form images on a single plane. Thedigital camera achieved in the embodiment generates an image that is areplication of a subject image formed at a specific image formingposition desired by the user, assumed along the optical axis.

In addition, the digital camera in the embodiment adopts such astructure that the image it generates is a synthetic image with a higherresolution than that matching the quantity of micro-lenses disposed inthe micro-lens array. Namely, a plurality of image-capturing pixels(cardinal-point pixels), which output image signals to be used forgeneration of individual picture elements constituting the syntheticimage, are disposed in correspondence to each micro-lens. The digitalcamera creates a synthetic image with an adjustable focus position so asto provide a synthetic image assuming a focus position selected by theuser, by adding image signals output from image-capturing pixels,disposed near a cardinal-point pixel, to the image signal output fromthe cardinal-point pixel thereby generating a synthetic image signalcorresponding to an image forming area equivalent to a single pixel inthe synthetic image. The following is a detailed description of theembodiment.

FIG. 1 shows the structure adopted in the digital camera achieved in theembodiment. The digital camera 1 allows an exchangeable lens 2, whichincludes a photographic lens L1, to be detachably mounted thereat. Thedigital camera 1 includes an image-capturing unit 100, a control circuit101, an A/D conversion circuit 102, a memory 103, an operation unit 108,a display unit 109, an LCD drive circuit 110 and a memory card interface111. The image-capturing unit 100 includes a micro-lens array 12achieved by disposing numerous micro-lenses 120 in a two-dimensionalarray, and an image sensor 13. It is to be noted that the followingdescription is given by assuming that a z-axis extends parallel to theoptical axis of the photographic lens L1 and that an x-axis and a y-axisextend perpendicular to each other within a plane ranging perpendicularto the z-axis.

An image is formed with a light flux traveling from a subject at aposition near the focal plane of the photographic lens L1, constitutedwith a plurality of optical lens groups. It is to be noted that FIG. 1shows the photographic lens L as a single representative lens forpurposes of simplification. The micro-lens array 12 and the image sensor13 are disposed in this order in the vicinity of the focal plane of thephotographic lens L1. The image sensor 13 is constituted with a CCDimage sensor or a CMOS image sensor, equipped with a plurality ofphotoelectric conversion elements. The image sensor 13 captures asubject image formed on its image-capturing surface and outputsphotoelectric conversion signals (image signals) that correspond to thesubject image, to the A/D conversion circuit 102 under control executedby the control circuit 101. It is to be noted that the image-capturingunit 100 will be described in detail later.

The A/D conversion circuit 102 executes analog processing on the imagesignals output by the image sensor 13 and then converts the analog imagesignals to digital image signals. The control circuit 101 is constitutedwith a CPU, a memory and other peripheral circuits. Based upon a controlprogram, the control circuit 101 executes specific arithmetic operationsby using signals input thereto from various units constituting thedigital camera 1 and then outputs control signals for the individualunits in the digital camera 1 so as to control photographing operations.In addition, based upon an operation signal input thereto via theoperation unit 108 in response to an operation of an aperture numberinput button 108 a, the control circuit 101 sets a synthetic imageaperture number having been selected by the user, as described infurther detail later. The control circuit 101 further determines asynthetic image focus position based upon an operation signal inputthereto via the operation unit 108 in response to an operation of afocus position input button 108 b, as described in further detail later.

The control circuit 101 has functions fulfilled by a table creation unit105, an image integration unit 106 and an image standardization unit107. The table creation unit 105 creates a synthesis affiliated pixeltable based upon the synthetic image aperture number, which isdetermined in response to the operation of the aperture number inputbutton 108 a. The image integration unit 105 generates synthetic imagedata by using the image signals based upon the synthetic image focusposition, determined in response to the operation of the focus positioninput button 108 b, and in reference to the synthesis affiliated pixeltable created by the table creation unit 105. The image standardizationunit 107 corrects the synthetic image corresponding to the syntheticimage data having been generated by the image integration unit 107 so asto achieve an aspect ratio (the ratio of width to height) of 1:1 for thesynthetic image, as described later. It is to be noted that the tablecreation unit 105, the image integration unit 106 and the imagestandardization unit 107 will all be described in detail later.

The memory 103 is a volatile storage medium used to temporarily storethe image signals having been digitized via the A/D conversion circuit102, data currently undergoing image processing, image compressionprocessing or display image data creation processing, and data resultingfrom the image processing, the image compression processing or thedisplay image data creation processing. At the memory card interface111, a memory card 111 a can be detachably loaded. The memory cardinterface 111 is an interface circuit that writes image data into thememory card 111 a and reads out image data recorded in the memory card111 a as controlled by the control circuit 101. The memory card 111 a isa semiconductor memory card such as a compact flash (registeredtrademark) or an SD card.

The LCD drive circuit 110 drives the display unit 109 as instructed bythe control circuit 101. At the display unit 109, which may be, forinstance, a liquid crystal display unit, display data created by thecontrol circuit 101 based upon image data recorded in the memory card111 a are displayed in a reproduction mode. In addition, a menu screenthat allows various operation settings to be selected for the digitalcamera 1 is brought up on display at the display unit 109.

Upon sensing a user operation performed thereat, the operation unit 108outputs a specific operation signal corresponding to the user operationto the control circuit 101. The operation unit 108 includes the aperturenumber input button 108 a, the focus position input button 108 b, apower button, a shutter release button, buttons related to settingmenus, such as a setting menu display changeover button and a settingmenu OK button and the like. The user, wishing to enter a specificsynthetic image aperture number F, operates the aperture number inputbutton 108 a. As the user operates the aperture number input button 108a and a specific aperture number F is thus selected, the operation unit108 outputs a corresponding operation signal to the control circuit 101.The user, wishing to enter a specific synthetic image focus position y,operates the focus position input button 108 b. As the user operates thefocus position input button 108 b and a specific focus position y isthus selected, the operation unit 108 outputs a corresponding operationsignal to the control circuit 101.

Next, the structure of the image-capturing unit 100 is described indetail. As explained earlier, the image-capturing unit 100 comprises themicro-lens array 12 and the image sensor 13. The micro-lens array 12 isconstituted with a plurality of micro-lenses 120 disposed in atwo-dimensional pattern. At the image sensor 13, pixel clusters 130,each of which receives light having passed through a specific micro-lensamong the micro-lenses 120 mentioned above, are disposed with an arraypattern corresponding to the array pattern of the micro-lenses 120. Eachpixel cluster 130 is made up with a plurality of photoelectricconversion elements 131 (hereafter referred to as image-capturing pixels131) disposed in a two-dimensional pattern.

FIG. 2A is a plan view, taken over the xy plane, of the micro-lenses 120disposed in the micro-lens array 12. As shown in FIG. 2A, a plurality ofmicro-lenses 120, each assuming a hexagonal shape, are disposed in ahoneycomb pattern on the xy plane. It is to be noted that FIG. 2A onlyshows some of the micro-lenses 120 among the plurality of micro-lenses120 disposed at the micro-lens array 12. FIG. 2B illustrates thepositional relationship among the photographic lens L1, the micro-lensarray 12 and the image sensor 13, assumed along the optical axis (alongthe z-axis) of the photographic lens L1. As shown in FIG. 2B, the imagesensor 13 is disposed at a position set apart by a focal length f of themicro-lenses 120. In other words, each pixel cluster 130 made up with aplurality of image-capturing pixels 131 assumes a position set apartfrom the corresponding micro-lens 120 by the focal length f of themicro-lens 120. It is to be noted that FIG. 2B shows only some of theplurality of micro-lenses 120 disposed at the micro-lens array 12 andonly some of the plurality of pixel clusters 130 and the plurality ofimage-capturing pixels 131 disposed at the image sensor 13.

The image integration unit 106 creates synthetic image data by usingimage signals output from the image sensor 13 structured as describedabove. The image integration unit 106 combines an image signal(hereafter referred to as a cardinal-point signal) output from aspecific image-capturing pixel 131 (hereafter referred to as acardinal-point pixel 132 (see FIG. 3)) among the image-capturing pixels131 making up the pixel cluster 130 disposed in correspondence to agiven micro-lens 120, with image signals output from image-capturingpixels 131 included in the pixel cluster 130 disposed for the micro-lens120 corresponding to the cardinal-point pixel 132 and pixel clusters 130corresponding to micro-lenses 120 disposed nearby. The image integrationunit 106 generates synthetic image signal equivalent to a single pictureelement through this process. The image integration unit 106 executesthe processing described above for all the cardinal-point pixelscorresponding to each micro-lens 120 and generates synthetic image databy adding together the individual synthetic image signals thusgenerated.

The image integration unit 106 generates the synthetic image signals asdescribed above, by referencing the synthesis affiliated pixel tablecreated by the table creation unit 105. The synthesis affiliated pixeltable indicates the position at which each image-capturing pixel 131among the image-capturing pixels that output image signals to becombined with the cardinal-point signal, is disposed in a pixel cluster130 corresponding to a specific micro-lens 120. The processing executedby the image integration unit 106 to generate synthetic image signals byusing image signals output from the image-capturing pixels 131 and theprocessing executed by the table creation unit 105 to create thesynthesis affiliated pixel table are now described.

FIG. 3 shows cardinal-point pixels disposed in correspondence to eachmicro-lens 120, i.e., disposed in each pixel cluster 130. FIG. 3, too,shows only some micro-lenses 120 among the plurality of micro-lenses120. As FIG. 3 indicates, four cardinal-point pixels 132 a to 132 d aredisposed in correspondence to each micro-lens 120 in the embodiment. Byproviding a plurality of cardinal-point pixels 132 in correspondence toeach micro-lens, the area where the focus position cannot be adjustedcan be reduced. Namely, while the size of the area where the focusposition cannot be adjusted is ±2f (f represents the focal length of themicro-lens 120) when the pixel cluster includes a single cardinal-pointpixel 132, the area can be reduced to ±f at the smallest by providing aplurality of cardinal-point pixels 132. Furthermore, by disposing agreater quantity of cardinal-point pixels 132, the number of pixelsconstituting the synthetic image data can be increased. In the examplepresented in FIG. 3, the number of pixels constituting the syntheticimage data is four times the number of micro-lenses 120 disposed at themicro-lens array 12.

In FIG. 3, the cardinal-point pixel 132 a is disposed in correspondenceto a pseudo-optical axis of the micro-lens 120. It is to be noted thatthe embodiment is described by assuming that the term “pseudo-opticalaxis” refers to the point at which the center of a light flux enteringfrom the pupil of the photographic lens IA and the principle plane ofthe micro-lens 120 intersect each other. In the example presented inFIG. 3, the geometrical center of the micro-lens 120 matches thepseudo-optical axis of the micro-lens. The cardinal-point pixels 132 band 132 c are disposed near adjacent micro-lenses 120, whereas thecardinal-point pixel 132 d is disposed on a boundary with an adjacentmicro-lens 120. In addition, the micro-lens 120 corresponding to thecardinal-point pixels 132 will be referred to as a cardinal-pointmicro-lens 121 in the following description.

—Generation of Synthetic Image Signals—

First, the synthetic image generation principle applicable to asynthetic image generated when the subject image is formed at the vertexof a micro-lens 120, as shown in FIG. 4A, i.e., when the focal plane Sis present at the vertex of the micro-lens 120, is described. In thissituation, the light fluxes from the subject enter the image-capturingpixels 131 in the pixel cluster 130 disposed in correspondence to themicro-lens 120. The image integration unit 106 generates a syntheticimage signal corresponding to one picture element to be part of thesynthetic image data by integrating the image signals output from theshaded image-capturing pixels 131 among the image-capturing pixels 131in FIG. 4A. The image integration unit 106 generates the synthetic imagedata by executing this processing for all the pixel clusters 130 eachcorresponding to one of the micro-lenses 120.

Next, the synthetic image generation principle applicable to a syntheticimage signal generated for a subject image formed at a given focal plane(image forming plane) is described. If the focal plane S is set apartfrom the vertex of the micro-lens 120, light fluxes from the subjectenter a plurality of micro-lenses 120 corresponding to differentclusters, as shown in FIG. 4(b). For this reason, the image integrationunit 106 needs to generate a synthetic image signal by using imagesignals output from image-capturing pixels 131 disposed incorrespondence to micro-lenses 120 near the cardinal-point micro-lens121, as well. In the embodiment, a plurality of cardinal-point pixels132 are set in correspondence to each cardinal-point micro-lens 121. Inother words, cardinal-point pixels 132, assuming positions other thanthe position corresponding to the pseudo-optical axis of the micro-lens120, are included in the cluster.

The image integration unit 106 generates a synthetic image signalequivalent to a single picture element (an image forming area in thesynthetic image) to be part of the synthetic image data by integratingall the image signals output from the image-capturing pixels 131contained in an integration area determined in correspondence to thesynthetic image aperture number. It is to be noted that such anintegration area is a circular area with a diameter D. The diameter D ofthe integration area may be expressed as in (1) below, with Frepresenting the aperture number (the synthetic image data aperturenumber) determined in response to an operation of the aperture numberinput button 108 a and f representing the focal length of themicro-lenses 120.D=f/F  (1)

FIG. 5 shows the relationship between the integration area Rs and theimage-capturing pixels 131. As described above, the image integrationunit 106 integrates the image signals output from all theimage-capturing pixels 131 contained in the circular integration areaRs. In FIG. 5, the image-capturing pixels 131 that output image signalsto be integrated are shaded. Each micro-lens 120 is just one of numerouslenses constituting the micro-lens array 12. This means that theintegration area Rs cannot assume a diameter greater than the diameterthat individual micro-lenses 120 may assume within the confines of thearray pattern of the micro-lenses 120. Accordingly, the largest aperturenumber Fmax that can be taken in conjunction with the synthetic imagedata is expressed as in (2) below. It is to be noted that “s” inexpression (2) represents the length of a side of an image-capturingpixel 131. In addition, the smallest aperture number Fmin that can betaken in conjunction with the synthetic image data matches the F numberof the micro-lenses 120.Fmax=f/s  (2)

A synthetic image signal generated by the image integration unit 106 byintegrating the image signals output from the pixel cluster 130 thatincludes the cardinal point pixels 132, i.e., the integral value, isexpressed as in (3) below. It is to be noted that P in expression (3)represents the output value indicated in the image signal output from animage-capturing pixel 131. In addition, “i” in expression (3) indicatesan image-capturing pixel 131 included in the integration area Rscorresponding to the synthetic image aperture number F and “0” indicatesthe micro-lens 120 disposed in correspondence to the pixel cluster 130containing the cardinal-point pixels 132, i.e., the cardinal-pointmicro-lens 121.P=ΣiF·Pi,0  (3)

As described above, the image integration unit 106 executes theintegrating operation by using image signals output from image-capturingpixels 131 included in pixel clusters 130 corresponding to micro-lenses120 disposed near the cardinal-point micro-lens 121, as well. Namely,the image integration unit 106 integrates the output values indicated inthe pixel signals from all the image-capturing pixels 131 forming anaggregate F{i} of image-capturing pixels 131 contained in theintegration area Rs set in correspondence to the synthetic imageaperture number F, which includes the image-capturing pixels 131 set incorrespondence to nearby micro-lenses 120 as well as the image-capturingpixels 131 disposed in correspondence to the cardinal-point micro-lens121. The output value P, which is calculated through this process, isexpressed as in (4) below. It is to be noted that “t” in expression (4)represents a nearby micro-lens 120, which may be the cardinal-pointmicro-lens 121 itself.P=ΣiF·Pi,t  (4)

FIGS. 6A and 6B show relationships among the image-capturing pixels 131that output the image signals used by the image integration unit 106 togenerate a single synthetic image signal, the cardinal-point micro-lens121 and nearby micro-lenses 120 a through 120 f adjacent to thecardinal-point micro-lens 121. It is to be noted that the syntheticimage signal is generated in correspondence to the cardinal-point signaloutput from the cardinal-point pixel 132 a in the examples presented inFIGS. 6A and 6B. The dispersed image-capturing pixels 131, disposed incorrespondence to the cardinal-point micro-lens 121 and the adjacentmicro-lenses 120 a through 120 f, as shown in FIGS. 6A and 6B, are theplurality of image-capturing pixels 131 contained in the area defined incorrespondence to the synthetic image aperture number F, as shown inFIG. 5, i.e., the plurality of image-capturing pixels 131 contained inthe integration area Rs.

It is crucial to accurately determine the exact position assumed by eachimage-capturing pixel 131 that outputs the image signal to be integratedwith the cardinal-point signal, in the pixel cluster 130 correspondingto a specific micro-lens 120 when the image integration unit 106integrates image signals through the processing described above.Accordingly, a synthesis affiliated pixel table indicating how theindividual image-capturing pixels 131, each represented by “i” inexpressions (3) and (4), are disposed in correspondence to the specificmicro-lenses 120 a to 120 f, i.e., indicating how the individualimage-capturing pixels 131 are dispersed, is stored in a predeterminedstorage area. The image integration unit 106 generates the syntheticimage signal by referencing the synthesis affiliated pixel table. It isto be noted that such a synthesis affiliated pixel table may beexpressed as in (5) below.t=Td(i)  (5)

The following is a description of the principal based upon whichsynthesis pixel affiliation tables are created.

FIG. 10 shows light sections LFD of a light flux that departs a lightpoint LP and travels via the micro-lens array 12 to the light-receivingsurfaces of image-capturing pixels 131 where it is sliced off. As shownin FIG. 10, while the light flux LF having departed the light point LPwidens, the angle by which it widens is restricted by theimage-capturing lens L1 disposed at a preceding stage. For this reason,the light flux LF having entered a given micro-lens 120 is containedwithin the area covered by the particular micro-lens (although FIG. 10shows light sections LFDc and LFDe appearing as if they each rangedbeyond the area covered by the corresponding micro-lens). This can besubstantiated by the fact that the light-receiving surfaces of theimage-capturing pixels 131 are set optically conjugate with the pupil ofthe image-capturing lens L1. When capturing an image via theimage-capturing lens L1, a photographic pupil image, i.e., a lightboundary, is formed within the area covered by each micro-lens 120 andthus, the light flux LF does not enter the area beyond the area coveredby the micro-lens 120.

The following explanation is provided on the premise outlined above. Thetotal quantity of light LF radiated on the micro-lens array 12 shown inFIG. 10, the widening angle of which is restricted by the pupil of thephotographic lens L1, in the light flux LF originating from the lightpoint LP, can be calculated by determining the cumulative value of thequantities of light entering image-capturing pixels 131 a through 131 ecorresponding to light sections LFDa to LFDe (generically referred to asan LFD) of the light flux LF. Accordingly, the image integration unit105 obtaining an image signal through integration, needs to determinethrough arithmetic operation LFD light sections at the light-receivingsurfaces of the image-capturing pixels 131 corresponding to thecoordinate value assumed by the light point LP along the z-axis. The“light point LP” viewed from the opposite side can be regarded as aconvergence point of a light flux LF emitted from display elements eachcorresponding to a light section LFD of the light flux LF and advancingas if to retrace the path through which the light flux enters theimage-capturing pixels as described above.

As explained earlier, the angle indicating the extent by which the lightflux LF departing the light point LP widens is determined by the pupilof the photographic lens L1, i.e., by the F number of theimage-capturing lens L1. It is to be noted that in a system without animage-capturing lens L1 such as a display system, the maximum aperture(smallest F number) is defined in correspondence to the F number of themicro-lenses 120. Accordingly, the aperture can be restricted simply byutilizing only a central portion of the area covered by each micro-lens120.

In reference to FIG. 11, showing light fluxes LF originating from lightpoints LP projected onto micro-lenses 120 as widened light fluxes, aspecific correspondence between micro-lenses 120 and light sections LFDare described. It is to be noted that FIG. 11 shows micro-lenses 120disposed in a square grid array so as to facilitate the explanation. Inaddition, FIG. 11 shows light fluxes LF widening from two differentlight points LP; a light point LP assuming a position along the z axisthat matches the focal length f of the micro-lenses 120 and a lightpoint LP assuming a position along the z-axis matching twice the focallength, i.e., 2f. In FIG. 11, the widened light flux LF departing thelight point LP set at the position f is indicated by a dotted line,whereas the widened light flux LF departing the light point LP assumingthe position 2f is indicated by a one-point chain line. The extent bywhich the light flux LF departing the light point LP assuming theposition matching the focal length f of a micro-lens 120 widens, isdefined by the micro-lens 120 (while the figure shows a circular lightsection LFD, a light section LFD will take on a square shape if themicro-lens 120 is optically effective through the corners of the square)and thus, the light flux LF enters the single micro-lens 120. Themicro-lens 120 corresponding to the particular light point LP is thusdetermined.

As long as the position of the light point LP matches the focal length fof a micro-lens 120, the light flux LF departing the light point LPwidens as a cone of light over the entire area directly under theparticular micro-lens 120. Accordingly, the image signals output fromall the image-capturing pixels 131 contained in the inscribed circlewithin the square area should be selected. If the absolute valueindicating the position assumed by the light point LP is less than thefocal length f, the light flux LF will widen instead of convergingwithin the area directly under the micro-lens 120. However, since theangle by which the incident light flux LF is allowed to widen isrestricted, the light section LFD is contained within the area coveredby the micro-lens 120.

The light flux departing the light point LP assuming the position 2f isdescribed next. FIG. 12 shows the micro-lenses 120 relevant to thislight flux. As shown in FIG. 12A, the relevant micro-lenses 120 includethe subject micro-lens 120, i.e., the cardinal point micro-lens 121, andthe eight micro-lenses 120 surrounding the cardinal point micro-lens121. Assuming that the opening area is restricted by the individualmicro-lenses 120, light sections LFD are bound to be present within thecovered areas i.e., the areas covered by the micro-lenses, which areshaded in FIG. 12A. In this situation, the light flux is sliced off overlight sections LFD, which are indicated as shaded areas in FIG. 12B atthe various micro-lenses 120.

As shown in FIG. 12B, the covered area corresponding to the singlecardinal point micro-lens 121 is divided and the divided areas aredistributed among the surrounding micro-lenses 120. The whole areaachieved by adding up the divided covered areas (partitioned areas)distributed among the neighboring micro-lenses is equivalent to theopening area of a single micro-lens 120. This means that the areal sizerepresenting the whole area of the light sections LFD corresponding to alight flux departing a light point LP remains uniform regardless of theposition of the light point LP. Accordingly, the total area representingthe sum of the partial areas can be calculated by simply determining thespecific micro-lens 120 from which the individual partial areasoriginate.

While FIG. 11 indicates the relationship between the position of thelight point LP and the magnification factor, i.e., the quantity ofmicro-lenses 120 present next to the cardinal point micro-lens 120, thisrelationship is applicable in a virtual opening area. In the embodiment,the opening area is divided in correspondence to a cluster ofmicro-lenses 120, reduced based upon the magnification factor, and splitpieces of the opening area are set at the corresponding positions withinthe micro-lenses 120 thus defined. The following description is given onan example in which the square containing opening area is reduced by anextent equivalent to a magnification factor of 2 and the opening area isthen divided (area division is applied) in correspondence to the arraypattern assumed for the micro-lenses 120.

FIG. 13 shows the light sections LFD resulting from the area divisiondescribed above are set around the cardinal point micro-lens 121. As theopening area is divided in this manner in correspondence to a specificmagnification factor, i.e., a light section LFD pattern corresponding tothe specific light point LP, is obtained. In more specific terms, theopening area is divided into a lattice having a width g/m, with “g”representing the diameter of the micro-lenses 120 (the length of eachside of a micro-lens). The magnification factor can be expressed as theratio m=y/f of the height (position) “y” of the light point LP and thefocal length f of the micro-lens. The ratio m may take a negative value.When the ratio “m” assumes a negative value, the light point LP may beregarded as being present further toward the image sensor 13 rather thantoward the micro-lenses 120.

While it is assumed that the light point LP is present on thepseudo-optical axis along the central axis of a given micro-lens 120 inthe description of the example provided above, the calculation describedabove can be executed without any problem even if the position of thelight point LP is offset from the pseudo-optical axis. If the arithmeticoperation could not be executed unless the light point LP was setexactly at the center of the lens, the two-dimensional resolution of thesynthetic image would be equal to the quantity of the micro-lenses 120,which would prove to be completely inadequate under normalcircumstances. For instance, in a configuration in which 100image-capturing pixels 131 are covered by each micro-lens 120, theresolution of the synthetic image would only be 1/100 of the quantity ofthe image-capturing pixels. In such a case, 100,000,000 image-capturingpixels 131 would be required in order to generate a synthetic imageexpressed with 1,000,000 pixels. Accordingly, image synthesis isexecuted at offset positions as well so as to allow a plurality of lightpoints LP to correspond to a single micro-lens 120.

The product of the area covered by each micro-lens 120 and the quantityof micro-lenses 120 is substantially equal to the total quantity ofimage-capturing pixels 131 and thus, generating synthetic image data bydesignating each of a plurality of off-centered points within a givenmicro-lens 120 as a cardinal point is equivalent to superimposing theimage signals from different image-capturing pixels 131. Namely, lightfluxes LF having departed the individual off-centered light points LPare superimposed upon one another at each image-capturing pixel 131.However, if the magnification factor is 1, this arithmetic operationwill result in simple interpolation, which does not contribute to anyimprovement in the resolution. This means that when an image is formednear the apex of the micro-lens 120, optical information expressingoptical depth will be lost.

FIG. 14 shows divided areas corresponding to a light point LPoff-centered to the left relative to the pseudo-optical axis of thecardinal point micro-lens 121. The height (position) of the light pointLP, which is offset from the center of the cardinal point micro-lens 121(with the lens diameter g), i.e., off-centered from the pseudo-opticalaxis, to the left by an extent p, is 2f. It is to be noted that a pointO1 and a point O2 in FIG. 14 respectively indicate the off-centeredlight point LP and the pseudo-optical axis. In this situation, thedivided areas shown in FIG. 14 are defined by shifting the micro-lenses120 in FIG. 13 to the right by the extent p and then dividing theopening area.

For instance, a micro-lens 120 centered on its pseudo-optical axis setat coordinates (0, 0) may be split into 16 portions with cut-offpositions taken at −g/2, −g/4, 0, g/4 and g/2 both along the x-axis andthe y-axis. By determining the individual divided areas accordingly andcalculating the total of all the divided areas, a group of 16 lightpoints can be obtained in correspondence to the single micro-lens 120.

—Synthesis Affiliated Pixel Table Creation Processing—

The image integration unit 106 references the synthesis affiliated pixeltable when it integrates image signals. As described earlier, thesynthesis affiliated pixel table makes it possible to ascertain thespecific position assumed by each image-capturing pixel 131 that outputsan image signal to be combined with the cardinal-point signal in a pixelcluster 130 which may correspond to the cardinal-point micro-lens 121 ora micro-lens 120 disposed near by the cardinal-point micro-lens 121.

Once the synthetic image focus position y and the synthetic imageaperture number F (field depth) are determined, the table creation unit105 creates a synthesis affiliated pixel table pertaining to theimage-capturing pixels 131 that output the image signals to be combinedwith the cardinal-point signal. As explained earlier, specificimage-capturing pixels 131 corresponding to specific micro-lenses 120,the image signals output from which are to be integrated with thecardinal-point signal, are determined in correspondence to the syntheticimage focus position.

FIG. 6A shows a relationship that may be assumed when the focus position(focal plane) y of the synthetic image is further toward the subjectrelative to the micro-lens array 12. FIG. 6B, on the other hand, shows arelationship that may be assumed when the focus position (focal plane) yof the synthetic image is further toward the image sensor 13 relative tothe micro-lens array 12. As FIG. 6A and FIG. 6B indicate, among theimage-capturing pixels 131 corresponding to the micro-lens 120 a,image-capturing pixels 131 taking up different positions are designatedas the image-capturing pixels 131 that output the image signals to beincorporated with the cardinal-point signal in correspondence to theposition of the focal plane. Image signals output from theimage-capturing pixels taking up different positions are used in theintegrating operation in correspondence to the other micro-lenses 120 bthrough 120 f and the cardinal-point micro-lens 121, as well.

If a plurality of cardinal-point pixels 132 (e.g., the cardinal-pointpixels 132 a through 132 d shown in FIG. 3) are present incorrespondence to the cardinal-point micro-lens 121, the table creationunit 105 creates a synthesis affiliated pixel table for each of thecardinal-point pixels 132. However, if a plurality of cardinal-pointpixels 132 are set in symmetry relative to the pseudo-optical axis ofthe cardinal-point micro-lens 121, the image integration unit 106 isable to utilize the synthesis affiliated pixel table created for onecardinal-point pixel 132 as the synthesis affiliated pixel table foranother cardinal-point pixel 132. For instance, the image integrationunit 106 is able to utilize the synthesis affiliated pixel table createdfor the cardinal-point pixel 132 b in FIG. 3 as the synthesis affiliatedpixel table for the cardinal-point pixel 132 d, which achieves symmetrywith the cardinal-point pixel 132 b relative to the pseudo-optical axisof the cardinal-point micro-lens 121.

The following is a detailed description of the synthesis affiliatedpixel table creation processing executed by the table creation unit 105.The following explanation focuses on the creation processing executed tocreate the synthesis affiliated pixel table for a typical cardinal-pointpixel, i.e., the cardinal-point pixel 132 a, which is disposed incorrespondence to the pseudo-optical axis of the cardinal-pointmicro-lens 121. The description is given by assuming that the focalplane of the synthetic image takes a position set apart from themicro-lens array 12 by a distance y, i.e., the focal length is y. Inaddition, a light flux passing through the pseudo-optical axis of an nthmicro-lens 120 taking up the nth position relative to the position ofthe cardinal-point micro-lens 121 enters at a position set apart by adistance x from the pseudo-optical axis of the cardinal-point micro-lens121, as expressed in (6) below. It is to be noted that “d” representsthe array pitch with which the individual micro-lenses 120 are disposed.x=fnd/y  (6)

Considering that the image-capturing pixels 131 each receive a lightflux with which an image is formed via the corresponding micro-lens 120,the width L of light included in the subject light from the focusposition y of the synthetic image, which is radiated via each micro-lens120 at the image-capturing plane of the image sensors 13, can beexpressed as in (7) below.1=fd/y  (7)

The light width l of the light represents a ring-shaped area (hereafterreferred to as a zone) assuming a width l on the two-dimensional planeof the image sensor 13. This means that a light flux defined incorrespondence to the synthetic image aperture number F enters the areadefined by the zone 1 at the micro-lens 120 assuming the nth positioncounting from the cardinal-point micro-lens 121. As expression (7)indicates, the zone 1 assumes a smaller width as the value representingthe synthetic image focus position y increases.

As indicated in FIG. 3, the micro-lenses 120 take on a hexagonal shapeover the xy plane and the hexagonal micro-lenses are disposed in ahoneycomb pattern in the micro-lens array 12 in the embodiment. FIG. 7shows a zone 11 defined when n=1 and a zone 12 when n=2 in theintegration area Rs corresponding to a given synthetic image aperturenumber F. As FIG. 7 indicates, the zone 11 corresponding to n assumingthe value of 1 is partitioned by the cardinal-point micro-lens 121 andthe micro-lenses 120 a to 120 f and thus, partitioned areas Rpa to Rpgare formed. Namely, the individual partitioned areas Rpa to Rpg arecovered by different micro-lenses 120. Accordingly, the imageintegration unit 106 calculates output values Pi, s indicated in theimage signals output from the image-capturing pixels 131 contained inthe various partitioned areas Rpa to Rpg within the zone 11. The imageintegration unit 106 then simply needs to integrate the output signalsin correspondence to the whole integration area Rs, i.e., all the zones1.

The relationships of micro-lenses 120 that are adjacent to one anotheramong the cardinal-point micro-lens 121 and the various micro-lenses 120a to 120 f are basically similar. Accordingly, the table creation unit105 determines the specific partitioned area Rp where a givenimage-capturing pixel 131, among the image-capturing pixels 131contained in the partitioned areas Rpa to Rpg constituting the zone 11,is present.

The diameter of the integration area Rs containing the image-capturingpixels 131 that output the image signals to be integrated with the imagesignal output from the cardinal-point pixel 132 a is expressed as D=f/F.In addition, it is assumed that the array pitch d with which themicro-lenses 120 are disposed along the x-axis (along the horizontaldirection), i.e., the diameter of the circle circumscribing eachhexagonal micro-lens 120, is equal to the maximum value Dmax that thediameter of the integration area Rs is allowed to take. In addition, thefocus position (focal length) assumed for the synthetic image is yrelative to the virtual plane of bend of the micro-lenses 120. In thissituation, the projected images formed by projecting the individualmicro-lenses 120 at the micro-lens array 12 onto the integration area Rsby magnifying their array pitch d by a projection magnification factorof fly, are each equivalent to one of the partition areas Rp defined inthe zone 1, as the individual micro-lenses 120 partition it. The tablecreation unit 105 generates a synthesis affiliated pixel table for thecardinal-point pixel 132 a, which indicates the correspondence betweenthe position of an image-capturing pixel 131 contained in eachpartitioned area Rp to correspondence between the position of animage-capturing pixel 131 contained in each partitioned area Rp and themicro-lens 120 that corresponds to the particular partitioned area Rp.The table creation unit 105 likewise generates synthesis affiliatedpixel tables for the other cardinal-point pixels 132 b to 132 d. It isto be noted that the position of the micro-lens 120 corresponding to aspecific partition area Rp is determined as a relative position inreference to the position of the cardinal-point micro-lens 121. It is tobe noted that generates a synthesis affiliated pixel table in a similarmanner for a cardinal-point pixel 132 that is not disposed on thepseudo-optical axis of the cardinal-point micro-lens 120, i.e., for eachof the cardinal-point pixels 132 b to 132 d, as well. For instance, acardinal-point pixel such as the cardinal point 132 c may be disposed onthe boundary of two micro-lenses 120 with an offset from thepseudo-optical axis of the cardinal-point micro-lens 121. In such acase, the table creation unit 105 projects the micro-lenses 120 onto theintegration area Rs so as to project them centered around thecardinal-point pixel 132 c decentered from the pseudo-optical axis ofthe cardinal-point micro-lens 121 by either increasing or decreasing thearray pitch d of the micro-lenses 120 by an extent matching theprojection magnification factor, as shown in FIG. 8.

It is to be noted that there should be a proportional relationshipbetween the positions of the micro-lenses 120 and the image-capturingpixels 131, i.e., between the size of the micro-lenses 120 (array pitchwith which the micro-lenses 120 are disposed) and the array pitch of theimage-capturing pixels 131. In conjunction with an array pitch set forthe image-capturing pixels 131 so that an inner dimension of themicro-lenses 120 is an integral multiple of the array pitch of theimage-capturing pixels 131, as shown in FIG. 3, the table creation unit105 is able to create synthesis affiliated pixel tables by repeatedlyexecuting identical arithmetic operations in correspondence to theindividual micro-lenses 120.

The operations executed in the digital camera 1 described above are nowexplained. As an operation signal is output from the operation unit 108in response to a shutter release switch operation by the user, thecontrol circuit 101 engages the image sensor 13 so as to start capturinga subject image and the image sensor outputs the resulting imagesignals. The image signals output from the image sensor 13 are convertedto digital image signals at the A/D conversion circuit 102 and thedigital image signals are stored into the memory 103. The imageintegration unit 106 generates synthetic image data by using the imagesignals stored in the memory 103.

Once the control circuit 101 determines the synthetic image aperturenumber F and the synthetic image focus position y in response tooperations of the aperture number input button 108 a and the focusposition input button 108 b performed by the user, the table creationunit 105 creates synthesis affiliated pixel tables, as has beendescribed earlier. Since the integration area Rs is determined incorrespondence to the synthetic image aperture number F, the tablecreation unit 105 calculates the projection magnification factor basedupon the synthetic image focus position y having been determined andthen defines the partitioned areas Rp within the integration area Rs.The table creation unit 105 creates a synthesis affiliated pixel tablebased upon the image-capturing pixels 131 contained in the partitionedarea Rp having been defined and then stores the synthesis affiliatedpixel table into a predetermined storage area.

The image integration unit 106 executes synthetic image data generationprocessing for the image signals stored in the memory 103 by using thesynthesis affiliated pixel tables thus stored in the storage area. Atthis time, the image integration unit 106 may execute the syntheticimage data generation processing by generating synthetic image signalsfor one pixel cluster 130 at a time among the pixel clusters 130, eachcorresponding to a specific micro-lens 120, or it may generate thesynthetic image data by sequentially generating synthetic image signalsin the order matching the order with which the individual cardinal-pointpixels 132 are disposed.

More specifically, the image integration unit 106 selects acardinal-point micro-lens 121 among the micro-lenses 120 and determinesthe integration area Rs centered on the pseudo-optical axis of thecardinal-point micro-lens 121. As explained earlier, the term“pseudo-optical axis” is used in the description of the embodiment torefer to the point at which the center of a light flux entering from thepupil of the photographic lens L1 and the principal plane of themicro-lens 120 intersect each other. Under normal circumstances, thepseudo-optical axis of a micro-lens 120 present further toward theperiphery of the micro-lens array 12, i.e., a micro lens 120 disposed incorrespondence to a position at which the subject image assumes agreater height, manifests a greater extent of offset relative to thegeometrical center of the micro-lens 120. Accordingly, in conjunctionwith the photographic lens L1 with the pupil thereof set at a fixedposition, the micro-lenses 120 should be designed so as to achieve aproportional relationship between the individual image-capturing pixels131 at the image sensor 13 and the pseudo-optical axis of the micro-lens120.

If the image-capturing pixels 131 do not have a proportionalrelationship to the pseudo-optical axes of the correspondingmicro-lenses 120, the table creation unit 105 needs to create synthesisaffiliated pixel tables through standardization by interpolating thepositions of image-capturing pixels 131 in correspondence to eachmicro-lens 120 in reference to the pseudo-optical axis of themicro-lens. Such standardization may be achieved by adopting, forinstance, the bicubic method or the nearest-neighbor method of the knownart.

The synthesis affiliated pixel tables are created by the table creationunit 105 based upon the positions of image-capturing pixels 131, whichare standardized as described above. Accordingly, the image integrationunit 106 is able to generate synthetic image data in reference to such asynthesis affiliated pixel table by integrating the image signals outputfrom image-capturing pixels 131 present within the integration area Rs,which is determined based upon the synthetic image aperture number F.The synthetic image data having been generated by the image integrationunit 106 then undergo standardization processing at the imagestandardization unit 107. The pitch of the synthetic image data havingbeen generated by the image integration unit 106 is determined basedupon the proportional relationship between the array pattern of themicro-lenses 120 and the array pattern of the image-capturing pixels131, and thus, the picture elements constituting the synthetic imagedata assume different pitch values along the horizontal direction andthe vertical direction. When the micro-lenses 120, assuming a hexagonalshape, are disposed in a honeycomb array and four cardinal-point pixels132 are disposed in each pixel cluster 130, as shown in FIG. 3, theratio of the horizontal pitch and the vertical pitch is calculated asexpressed in (8) below.1□:□0.8660=1□:□3/2  (8)

Accordingly, the image standardization unit 107 executes standardizationthrough interpolation operation executed along the vertical direction orthe horizontal direction by adopting, for instance, the nearest-neighbormethod or the bicubic method of the known art. As a result, the ratio ofthe horizontal pitch of the synthetic image data to the vertical pitchof the synthetic image data is modified to 1:1 via the imagestandardization unit 107.

The following advantages are achieved with the digital camera 1 in theembodiment described above.

(1) A plurality of micro-lenses 120 are disposed in a two-dimensionalarray near the focal plane of the photographic lens L1 and the imagesensor 13 includes a two-dimensional array of pixel clusters 130 andeach made up with a plurality of image-capturing pixels 131 disposed incorrespondence to one of the micro-lenses 120. The image-capturingpixels 131 in the pixel clusters 130 receive, via the micro-lenses 120,light fluxes from the subject, having passed through the photographiclens L1, and output image signals. The image integration unit 106generates synthetic image signals corresponding to a plurality of imageforming areas at a given focus position, i.e., on a given image formingplane assumed for the photographic lens L1, by combining image signalsoutput from specific image-capturing pixels 131 based upon a synthesisaffiliated pixel table indicating the specific positions of theimage-capturing pixels 131 that output image signals used for purposesof generating a synthetic image signal in each image forming area. Thetable creation unit 105 creates synthesis affiliated pixel tables incorrespondence to each focus position.

In the related art, a synthetic image signal equivalent to one pictureelements among the picture elements constituting synthetic image data ona given focal plane is generated by combining image signals output incorrespondence to the light fluxes having passed through a coordinatepoint that corresponds to the central position assumed by the micro-lens120. Namely, the number of picture elements constituting the syntheticimage data, i.e., the resolution of the synthetic image data matches thenumber of micro-lenses 120 set in the array. For instance, assuming thatthe diameter of each micro-lens 120 is equal to ten image-capturingpixels 131 set side-by-side, i.e., assuming that 10×10 image-capturingpixels 131 are arrayed together in correspondence to each micro-lens120, the image density of the image expressed with the synthetic imagedata generated in the camera 1 will be 1/100, since the number ofpicture elements constituting the synthetic image data is equal to thenumber of micro-lenses 120 disposed in the array. In other words, evenif 10,000,000 image-capturing pixels 131 are arrayed at the image sensor13, synthetic image data expressed with only 100,000 picture elementswill be generated.

The digital camera 1 achieved in the embodiment is distinguishable fromthose in the related art in that four cardinal-point pixels 132, incorrespondence to each of which a synthetic image signal equivalent toone pixel element in the synthetic image data is generated, are disposedin correspondence to a single micro-lens 120. As a result, betterresolution is assured over that of synthetic image data generated in adigital camera adopting the technologies of the related art. In otherwords, a higher level of resolution than that matching the number ofmicro-lenses 120 in the array is achieved, which makes it possible toprovide a synthetic image of better quality

(2) The table creation unit 105 creates a synthesis affiliated pixeltable indicating the positions of the image-capturing pixels 131 thatoutput image signals to be combined with the cardinal-point signaloutput from a cardinal-point pixel 132. This means that synthetic imagedata can be generated without having to execute complicated arithmeticoperations involving the Fourier transform, which the technologies inthe related art require. Consequently, even when the micro-lenses 120are not disposed in a rectangular array (e.g., even when themicro-lenses 120 are disposed in a honeycomb array) on the xy plane andthus the arithmetic operations involving the Fourier transform would beparticularly complicated, the synthetic image can be generated at highspeed while assuring a lighter processing load by adopting the presentinvention. Furthermore, since the table creation unit 105 createssynthesis affiliated pixel tables each time a specific synthetic imagefocus position y is selected, there is no need to store in advance anextremely large number of tables each in correspondence to a specificcardinal-point pixel 132 among a plurality of cardinal-point pixels 132at a specific synthetic image focus position y among a plurality ofsynthetic image focus positions and thus, the memory space can beutilized sparingly.

(3) The positional interval with which the plurality of cardinal-pointpixels 132 are disposed has a proportional relation to the array pitchwith which the plurality of micro-lenses 120 are disposed. In otherwords, the array pitch for the image-capturing pixels 131 is determinedso that the value representing the size of the micro-lenses 120 is anintegral multiple of the array pitch assumed for the image-capturingpixels 131. As a result, the image integration unit 106 is able togenerate synthetic image data for a cardinal-point pixel 132corresponding to a given micro-lens 120 by executing processingidentical to the synthetic image generation processing executed for acardinal-point pixel 132 corresponding to another micro lens 120 amongthe plurality of micro-lenses 120. Synthetic image data can thus begenerated at high speed by lessening the load of the synthetic imagegeneration processing.

(4) The table creation unit 105 creates a synthesis affiliated pixeltable in correspondence to each of the plurality of cardinal-pointpixels 132 a to 132 d. Such a synthesis affiliated pixel table indicatesthe positions of the image-capturing pixels 131 corresponding to thecardinal-point pixel 132, which are standardized in reference to thepseudo-optical axis of the micro-lens 120. In addition, the synthesisaffiliated pixel table specifies the relative position of themicro-lenses 120 corresponding to the positions at which theimage-capturing pixels 131 are disposed in reference to the position ofthe cardinal-point micro-lens 121. Consequently, the image integrationunit 106 is able to select the image-capturing pixels 131 that outputthe image signals to be combined with the cardinal point image signalprovided from the cardinal-point pixel 132 simply by referencing thesynthesis affiliated pixel table, thereby achieving faster syntheticimage data generation processing.

(5) The synthesis affiliated pixel tables created by the table creationunit 105 each indicate the correspondence of the position of animage-capturing pixel 131 contained in the integration area Rs setaround a cardinal pixel 132 designated as a cardinal point and assuminga diameter D (=f/F) equal to the value obtained by dividing the focallength f of the micro-lenses 120 by the synthetic image aperture numberF, to a specific micro-lens 120 among the plurality of micro-lenses 120.As a result, the image integration unit 106 is able to select the imagesignals to be combined with the cardinal-point signal provided from thecardinal-point pixel 132, i.e., the image signals corresponding to thesynthetic image aperture number F, simply by referencing the synthesisaffiliated pixel table, assuring faster synthetic image data generationprocessing.

(6) The image standardization unit 107 standardizes the synthetic imagedata generated by the image integration unit 106 so as to modify theratio of the horizontal pitch and the vertical pitch of the syntheticimage data to 1. Thus, even when the micro-lenses 120 assume a shapeother than a rectangle over the xy plane, synthetic image data withmatching horizontal and vertical pitches are generated so as to providea synthetic image with the highest possible image quality.

The digital camera 1 achieved in the embodiment as described above,allows for the following variations.

(1) The four cardinal-point pixels 132 a to 132 d disposed incorrespondence to each micro-lens 120 may assume positions decenteredfrom the pseudo-optical axis of the micro-lens 120. FIG. 9A presents anexample of positions that may be taken by such cardinal pixels 132. Thecardinal-point pixels 132 a to 132 d in this example are disposed atpositions so as to assume symmetry to one another relative to thepseudo-optical axis of the micro-lens 120. The image integration unit106 is thus able to utilize the synthesis affiliated pixel table createdin correspondence to a cardinal-point pixel 132 among the cardinal-pointpixels 132 disposed as shown in FIG. 9A in conjunction with the otherthree cardinal-point pixels 132, as well. For instance, the imageintegration unit 106 may utilize the synthesis affiliated pixel tablecreated in correspondence to the cardinal-point pixel 132 a as asynthesis affiliated pixel table for the cardinal-point pixel 132 b byrotating the initial synthesis affiliated pixel table by 60°, then as asynthesis affiliated pixel table for the cardinal-point pixel 132 e byfurther rotating the synthesis affiliated pixel table by 60° and as asynthesis affiliated pixel table for the cardinal-point pixel 132 d byrotating the synthesis affiliated pixel table by another 60°. In otherwords, the table creation unit 105 simply needs to create a singlesynthesis affiliated pixel table in correspondence to each micro-lens120.

(2) The present invention may be adopted in conjunction with, forinstance, square micro-lenses 120 disposed in a rectangular array,instead of the micro-lenses 120 assuming a hexagonal shape and disposedin a honeycomb array over the xy plane. FIG. 9B presents an example of apositional arrangement that may be adopted in conjunction with suchsquare micro-lenses 120 and the corresponding cardinal-point pixels 132.In this case, too, the table creation unit 105 will create synthesisaffiliated pixel tables each corresponding to one of the fourcardinal-point pixels 132 a to 132 d through a method similar to thatdescribed in reference to the embodiment and then the image integrationunit 106 will generate synthetic image data by referencing the synthesisaffiliated pixel tables. However, unlike the synthetic image datagenerated in conjunction with the micro-lenses 120 disposed in thehoneycomb array, the synthetic image data generated in conjunction withthe square micro-lenses will achieve a ratio of 1:1 for the horizontaldata pitch and the vertical data pitch, thereby eliminating the need forany standardization processing by the image standardization unit 107.

(3) While synthesis affiliated pixel tables are created in theembodiment each in correspondence to one of the four cardinal-pointpixels 132, the present invention is not limited to this example and n(n represents an integer equal to or greater than 1) synthesisaffiliated pixel tables may be created in correspondence to eachmicro-lens. For instance, if the pixel cluster 130 corresponding to thecardinal-point micro-lens 120 includes 16 cardinal-point pixels 132,i.e., when generating synthetic image data with a resolution 16 timesthe array pitch of the micro-lenses 120, the table creation unit 106will need to create 16 different synthesis affiliated pixel tables. Itis to be noted that if a single cardinal-point pixel 132 is disposed incorrespondence to the cardinal-point micro-lens 121, the resolution ofthe synthetic image data will match the number of micro-lenses 120disposed in the micro-lens array.

As long as the features characterizing the present invention are notcompromised, the present invention is not limited to the particulars ofthe embodiments described above and other modes that are conceivablewithin the technical scope of the present invention will also be withinthe scope of present invention. The embodiment and the variationsdescribed above may be adopted in any combination.

What is claimed is:
 1. An image processing device, comprising: an imagegenerating unit that, when executed by a processor, generates an imagein correspondence to a subject on a focal plane based upon output datafrom a plurality of light receiving units disposed on each of aplurality of micro-lenses, each of the plurality of light receivingunits is included in an image sensor; and a control circuit that causesthe image generating unit to generate a pixel image of the imagecorresponding to a first region of the subject, based upon the outputdata from the light receiving unit, into which light from the firstregion of the subject enters, among the light receiving units other thana light receiving unit on an optical axis of the first micro-lens amongthe plurality of micro-lenses and the output data from the lightreceiving unit, into which light from the first region of the subjectenters, disposed on a second micro-lens among the plurality ofmicro-lenses and to generate a pixel image of the image corresponding toa second region of the subject, based upon the output data from thelight receiving unit, into which light from the second region of thesubject enters, disposed on the optical axis of the first micro-lens andthe output data from the light receiving unit, into which light from thesecond region of the subject enters, disposed on a third micro-lensamong the plurality of micro-lenses, the second micro-lens and the thirdmicro-lens are positioned near the first micro-lens, and the lightreceiving unit, into which light from the first region of the subjectenters, among the light receiving units disposed on the secondmicro-lens and the light receiving unit, into which light from thesecond region of the subject enters, among the light receiving unitsdisposed on the third micro-lens are changed in case that the positionof the focal plane shifts.
 2. The image processing device according toclaim 1, wherein: the control circuit causes the image generating unitto generate the image in which a number of the pixels is larger than anumber of the plurality of micro-lenses.
 3. The image processing deviceaccording to claim 1, wherein the image generating unit generates, asthe image in correspondence to the subject on the focal plane, an imageof the subject on a focal plane of an optical system used for capturingthe image of the subject.
 4. The image processing device according toclaim 1, wherein the light receiving unit, into which light from thefirst region of the subject enters, among the light receiving unitsother than the light receiving unit on the optical axis of the firstmicro-lens is used regardless of a position of the focal plane.
 5. Theimage processing device according to claim 1, wherein the lightreceiving unit, into which light from the second region of the subjectenters, disposed on the optical axis of the first micro-lens is usedregardless of a position of the focal plane.
 6. The image processingdevice according to claim 1, wherein the output data from the lightreceiving units is stored in a storage medium, and the control circuitreads out the output data from the storage medium.
 7. Animaging-capturing device comprising the image processing deviceaccording to claim
 1. 8. The image processing device according to claim1, wherein the first region of the subject is located at a positiondifferent from a position of the second region of the subject on a planeorthogonal to the optical axis.